Movement detection device and focus detection apparatus using such device

ABSTRACT

A movement detection device for detecting movement of an image incident on an image sensing device and for compensating for that movement includes movement detection circuitry for detecting the movement of the image from image signals output from the image sensing device. Correction circuitry is provided for electrically correcting the movement of the image based on an output of the movement detection circuitry. A filter is provided for receiving the corrected image signals output from the correction circuitry, and for performing a filter processing operation on the corrected image signals to compensate for degradation of image resolution due to the movement of the image.

This application is a continuation of application Ser. No. 07/983,277filed Nov. 30, 1992, which is a continuation of Ser. No. 07/691,785,filed Apr. 26, 1991, both now abandoned.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a movement detection device suitablefor use with (i) a camera having a vibration-proof device forcompensating for the movement of an image being picked-up by the cameradue to trembling of the hand or other vibrations, (ii) an automatictracking apparatus for tracking a moving object, or (iii) an automaticfocus detection apparatus for detecting a focusing state from an imagepick-up signal.

2. Related Background Art

Recently, image instruments such as a video camera or electronic camerahave been remarkably developed in which, to enable a more reliable andappropriate photographing operation, a movement correction device hasbeen adopted which allows the photographing of higher quality imageswithout fluctuation by correcting the movement of the image due totrembling of the hand or other vibrations.

The movement correction method for the movement correction deviceinvolves (i) a mechanical correction method of using inertia to maintainthe axes of the lens and image sensor fixed against the rotation ofcamera body, (ii) an optical correction method of using an opticalmember such as a variable apex angle prism, and (iii) an imageprocessing correction method for making the correction by moving theimage with image processing.

According to the mechanical correction method, a special structure forsupporting the lens and a pick-up system is required. According to theoptical correction method, a special optical member such as a variableapex prism is required, while in the image processing correction method,no special mechanical structure or optical members are necessary. In theimage processing correction method, the movement is corrected only withsignal processing using electrical circuitry, and this method isexpected to be widely used in the future.

However, the movement correction device with the above-mentionedconventional image processing method has the following disadvantage,compared with the mechanical or optical device. That is, in performingthe movement correction with the image processing method, an image hassome movement at the pick-up stage (with an image sensor or pick-uptube), whereby in the post-pick-up processing, the movement of the imagewithin an image screen is removed by shifting the image in accordancewith the amount of image movement.

Thus, as the image obtained in the pick-up stage yields some unfocusedportions, final resolution of the image is low even if the movement ofthe image is corrected in the post-pick-up processing, so that a poorquality of image is output.

Recently, in video instruments such as a video camera or an electroniccamera, an automatic focusing adjustment apparatus for adjusting thefocus by detecting the focusing state from a pick-up signal has beenadopted, but as above described, the movement of the image may reduce ahigh frequency component varied with the focusing state from the pick-upsignal, thereby decreasing the sharpness, and degrading the performanceof automatic focusing adjustment apparatus, whereby there is a risk of amalfunction. Thus, it is quite important to detect and correct themovement of the image in the signal processing using the pick-up signal.

SUMMARY OF THE INVENTION

The present invention is intended to resolve the above problems, and itis a first object to provide a movement detection device which cancompensate for the degradation of resolution due to the movement of theimage.

A second object of the present invention is to provide a movementdetection device capable of providing an image without fluctuation ordeflection and having high resolution, with the movement corrected, andwhich can compensate for the degradation of resolution with the movementof the image by means of filtering.

A third object of the present invention is to provide a movementdetection device capable of providing a high quality image (with itsmovement corrected) by performing filtering adaptively by the use of amovement vector of the image obtained from a movement amount detectiondevice especially useful for the correction of movement.

A fourth object of the present invention is to provide a movementdetection device having the effect of providing a high quality image bycompensating for the degradation of image quality resulting from themovement of the image by means of filtering, and increasing theresolution of the output image for a movement correction device.

A fifth object of the present invention is to provide a movementdetection device in which the quality of the image can be madeexcellent, almost optimal, without an increase of the cost, by changingadaptively the characteristics of a filter used in the filteringprocessing, particularly based on a movement vector to be obtained forthe correction of movement.

To achieve such objects, a preferred embodiment of the present inventionfeatures a movement detection device for detecting the movement of animage and compensating for the movement of the image, comprisingmovement detection means for detecting a movement of an image,correction means for correcting the movement of the image based on anoutput of said detection means, and filter means for performing afiltering processing compensating for degradation of resolution with themovement of the image.

A sixth object of the present invention is to provide a focus detectionapparatus capable of making a high precision focus detection whileavoiding the decrease of accuracy due to the movement of the image,wherein the movement of the image is detected and the focus detection ismade using a signal for which the degradation of the resolution with themovement of the image has been compensated.

A seventh object of the present invention is to provide a stable focusdetection apparatus with high precision at all times, in which theapparatus is not subject to the influence of camera vibration or objectmovement because it is possible to prevent the degradation of accuracyin focus detection means caused by a signal decreasing in accordancewith the focusing state, such as unclearness of an edge portion or adecrease of the high frequency component owing to the movement of theimage.

To accomplish such objects, a preferred embodiment of the presentinvention features a focus detection apparatus for detecting a focusingstate based on a pick-up signal output from pick-up means, comprisingmovement detection means for detecting a movement of an image from thepick-up signal, movement correction means for correcting the movement ofthe image based on an output of the movement detection means, and focusdetection means for performing focus detection by extracting a signalcomponent varying with the focusing state from the pick-up signal havingits movement component corrected by the movement correction means.

Additional objects and feature of the present invention will becomeapparent from the following description and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a first example of a movementdetection device according to the present invention.

FIG. 2 is a view showing the movement of an image.

FIG. 3 is a view showing the movement of an optical image.

FIGS. 4A to 4C are characteristic views showing the filtercharacteristic for filtering.

FIG. 5 is a characteristic view showing the inverse filtercharacteristic.

FIG. 6 is a characteristic view showing the Wiener filtercharacteristic.

FIG. 7 is a block diagram showing a second example of the presentinvention.

FIG. 8 is a view showing images containing a plurality of movements.

FIG. 9 is a view showing area decision results.

FIG. 10 is a block diagram showing a third example of a movementcorrection device according to the present invention.

FIG. 11 is a view showing the movement of the image.

FIG. 12 is a view showing the movement of an optical system.

FIG. 13 is a view showing a point image distribution function with themovement of the image.

FIG. 14 is a view showing the correction of an edge width.

FIG. 15 is a block diagram showing a fourth example of the presentinvention.

FIGS. 16A and 16B are schematic graphs showing a focus detection methodusing a conventional image processing.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of a movement correction device according to the presentinvention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a first embodiment of a movementcorrection device according to the present invention.

In FIG. 1, 1 is an object, 2 is a pick-up lens, and 3 is a pick-upelement (or pick-up tube) such as a CCD for outputting a pick-up signalby photoelectrically converting an object image formed on a pick-upplane by the pick-up lens 2. 4 is an amplifier for amplifying thepick-up signal output from the pick-up element 3 to a predeterminedlevel, 5 is an A/D converter for converting the input analog pick-upsignal to a digital signal, 6 is a frame memory for storing the imagesignal converted into the digital signal by the A/D converter 5, and 7is a movement amount detection circuit for obtaining a movement vectorfrom the image signal which has been converted from analog to digitalform. Exemplary of a method of calculating the movement vector is thatmethod used in a so-called representative point matching method orgradient method. 8 is a memory reading circuit for reading an imagesignal form the frame memory 6 by generating a read address andexecuting the reading.

9 is a parameter setting circuit for setting parameters for preventionof image degradation, and 10 is a filter.

11 is a D/A converter for converting the digital image signal passingthrough the filter 10 to an analog image signal, 12 is a synchronismsignal adding circuit for adding a synchronism signal of the imagesignal, and 13 is an output video signal.

The pick-up leans 2 forms an image of object 1 on the pick-up plane ofpick-up element 3. The image on the pick-up element 3 includes movementsowing to the movement of the lens 2, pick-up element 3, or object 1. Theimage signal output from the pick-up element 3 is amplified by theamplifier 4, converted to the digital signal by the A/D converter 5, andthen stored in the frame memory 6, temporarily.

The digital image signal which is an output of A/D converter 5 is alsotransferred to the movement amount detection circuit 7. Movement vectordata obtained by the movement amount detection circuit 7 are transferredto the memory reading circuit 8 and the parameter setting circuit 9.

The movement detection circuit 7 requires data of an image screen of oneframe or one field prior to the current screen for calculating theamount of movement, thereby requiring a frame memory. This frame memorymay be constructed in common with the frame memory 6, or separatelyprovided.

The memory reading circuit 8 creates the address for reading the framememory 6 multiplied by the offset based on movement vector data.Thereby, data read from the frame memory 6 are read out in such a way soas to move data almost reversely to the movement of the image, so thatthe movement of the image can be corrected. That is, the correction ofimage deflection can be achieved in the memory.

The parameter setting circuit 9 determines parameters such as filtercoefficients based on the movement vector obtained from the movementamount detection circuit 7, and sends them to the filter 10. The filter10 filters the image signal having the corrected movement read from theframe memory 6, so as to reduce the unfocused part resulting from themovement of the image on the pick-up element 3, i.e., the degradation ofresolution.

Also, the filter 10 has the characteristics of a high-pass filter and aband-pass filter, as will be described later.

The image signal output from the filter 10 is converted to an analogsignal by the D/A converter 11, synthesized with a synchronism signal bythe synchronism signal adding circuit 12, and output as the video signal13.

The improvement of resolution with the filtering will be described inthe following.

FIG. 2 is a plan view showing the movement of an image.

21 is an output screen. The output screen 21 has a fixed coordinatesystem which is useful as a reference to perform the processing such asvibration isolation. Here, a monitor display screen can be assumed, forexample.

22 is a previous image, and 23 is a current image. 24 is a movementvector when the previous image 22 moves to the current image 23.

In the output screen 21, a video signal 13 of FIG. 1 is displayed. Forexample, an image displayed with trembling of the hand shows themovement from previous image 22 to current image 23 if time has passedfrom one clock to another when the object 1 is stationary.

With the movement correction method in the image processing, themovement correction is performed in such a manner that the movementvector 24 is calculated in the movement amount detection circuit 7, datais shifted by the amount of movement vector 24 on the output screen 21when date of current image 23 is read from the frame memory 6, and theoffset is added to the read address so that current image 23 can besuperimposed almost on previous image 22.

As current image 23 includes movement, the value of each pixel isintegrated in a direction of movement vector 24. Accordingly, inpractice, current image 23 of FIG. 2 only shows approximately agravitational center position of sides of each pattern.

FIG. 3 is a front view showing the movement of the optical system. Inthe same figure, 31 is one cell when the pick-up element is asolid-state pick-up element such as a CCD. For the convenience ofexplanation, it is assumed that the movement direction of the imagecoincides with a direction of the array for one cell in the pick-upelement 3. The pick-up element 3 converts a pattern of the optical imagethereon during a predetermined period of exposure. 32 is an opticalimage at the start of exposure, and 33 is an optical image at thetermination of exposure. The optical image 32 at the start of exposureis moved to the optical image 33 at the termination of exposure becausethe image moves during exposure as shown in FIG. 2.

The movement vector 24 as shown in FIG. 3 indicates the movement betweenthe optical image 32 at the start of exposure and the optical image 33at the termination of exposure. This is almost the same as that of FIG.2, but more strictly, slightly different. That is, the movement vectoras shown in FIG. 2 is one taken at nearly intermediate times ofrespective exposure periods for two images. On the contrary, themovement vector as shown in FIG. 3 is one taken for an image of interestfrom the start of exposure to the termination of exposure. Accordingly,when the amount of image movement changes abruptly, both movementvectors will have different values. Generally, both of the values can bealmost the same, but the movement vector 24 as shown in FIG. 3 may beused after some slight correction when required in the parameter settingcircuit 9, because the movement vector 24 obtained from the movementdetection circuit 7 is one as shown in FIG. 2.

FIGS. 4A-4C are graphs showing the filter characteristics for thefiltering in the filter 10 for correcting the degradation of the image.

In FIG. 4A, 41 is a point image distribution function with the movementof the image. The axis of abscissa x in the spatial coordinate system istaken along the movement direction. The length of the movement vector 24is assumed to be a. Then, the image generated by one point of object 1moves during the exposure period, approximately following the pointimage distribution function 41.

Representing that function as h(x)

    h(x)=Rect(x/a)                                             (1)

42 in FIG. 4B is a frequency characteristic showing the degradation ofthe image.

Representing the frequency as f and the frequency characteristic 42 asH(f),

    H(f)=(sinπaf)/πf                                     (2)

because H(f) is a Fourier transform of the point image distributionfunction h(x).

43 in FIG. 4C is a frequency characteristic of the filter, andrepresenting it as P(f), ##EQU1##

Such a filter is called as an inverse filter.

That is, H(f)·P(f)=1, which means that the filtering of P(f) cancompensate for the degradation of the image with H(f).

By filtering with filter 10, the resolution of output video signal 13 isenhanced and an excellent quality image can be obtained. However, theinverse filter can be realized only approximately because it hasinfinite value at the frequency where the Fourier transform H(f) iszero, and the frequency range can be a range where the frequencyspectrum exists.

As clearly seen from FIGS. 4A-4C and expressions (2) and (3), thecharacteristic of degradation H(f) and the characteristic ofcompensation filter P(f) contain the size a of movement vector 24 as aparameter. The x-axis and f-axis are each taken along the direction ofmovement vector 24, which reveals that P(f) depends on the size anddirection of movement vector 24.

Accordingly, it is desirable that the filter 10 has its filtercharacteristic changed adaptively depending on the movement vector 24.

The filter 10 has two methods of performing the filtering processing.One of them is filtering on the axis of frequency, in which the Fouriertransform is taken of the image signal read out from the frame memory 6with an FFT (Fast Fourier Transform), which is then multiplied by theinverse filter P(f), and the inverse Fourier transform is taken so thata filtered image signal is obtained.

Another filtering method is filtering on the axis of time, performed insuch a way that the impulse response is obtained by the inverse Fouriertransformation of inverse filter P(f), and convoluted to the imagesignal from the frame memory 6 so that a filtered output can beobtained.

When the inverse filter is realized with the convolution in the axis oftime, an impulse response S(x) of the filter in the following expressioncan be used. ##EQU2## Where K is a proportional constant, δ(x) is adelta function, and δ'(x) is a derivative of the delta function. And "*"is a symbol indicating the convolution, and sign(x) indicates a signsuch that ##EQU3## Note that the expression (4) is obtained by takingthe Fourier transform of expression (3) using the delta function.

FIG. 5 is a graph showing the impulse response of an inverse filter.

51 is an impulse response of the inverse filter, representing theexpression (4).

The expression (4) must be truncated midway because it continuesinfinitely in the direction of the x-axis. Therefore, it is desirable touse that function in the filtering processing after multiplication ofthe window function such as a hamming window.

In the inverse filter, the frequency area where the image information isalmost lost with the degradation of the image, i.e., where the value offrequency characteristic 42 is zero, and the high frequency area wherethere is only a little image information by nature may be given thecharacteristic of a large gain so that the output image often has a poorS/N ratio. Therefore, in the filter 10, the Wiener filter can besubstituted.

The frequency characteristic of the Wiener filter R(f) can berepresented in the following expression. ##EQU4## Where Φ_(n) (f) andΦ_(s) (f) indicate the power spectrum of noise and image signal,respectively, and * indicates the complex conjugate.

Here, as it is difficult to obtain Φ_(n) (f) and Φ_(s) (f) correctly,Φ_(n) (f) is set to be constant by assuming white noise, and Φ_(s) (f)is set to be the Gaussian type. Or they can be predetermined with theassumption that Φ_(n) (f)/Φ_(s) (f) is constant over all frequencies.

In the Wiener filter, at the frequency where the signal component issufficiently larger than the noise component, the value is almost thesame as that of the inverse filter, or conversely, it is close to zeroat the frequency where the noise component is larger than the signalcomponent.

FIG. 6 is a graph showing the frequency characteristic of a Wienerfilter. 61 is a frequency characteristic of the Wiener filter. Comparedwith the frequency characteristic 43 of the inverse filter, it can beseen that the gain is smaller at the frequency area with the poor S/Nratio. And in the Wiener filter, like the inverse filter, it isdesirable that the characteristic may be adaptively changed depending onthe movement vector 24.

Various filter characteristics of the filter 10 have generally thecharacteristic of a high-pass filter or a band-pass filter.

FIG. 7 is a block diagram showing a second embodiment of the presentinvention.

The second embodiment shows a device which is effective when there are amoving area and a stationary area in an image, and further when themoving area is separated into a plurality of regions which havedifferent movement vectors.

71 is an output of the movement amount detection circuit 7, i.e., themovement vector for each block or pixel within a screen. 72 is an areadiscrimination circuit, 73 is its output, i.e., an address offsetsignal, 74 and 75 are other outputs of the area discrimination circuit72, i.e., the area signal and movement vector within the area,respectively. 76 is a switch for sending the input signal to either oftwo output lines. 77 is an output video signal being processed for eacharea.

The stages where the input image signal is converted from analog todigital form, stored into the frame memory 6, and transferred to themovement amount detection circuit 7 are the same as in the firstembodiment.

The movement amount detection circuit 7 transfers the movement vector 71to the area discrimination circuit 72.

The area discrimination circuit 72 divides the screen into thestationary area and a plurality of moving areas having differentmovement vectors, based on the movement vector 71. The areadiscrimination circuit 72 selects a desired area of divided areas, andsends the movement vector of the selected area as an address offsetsignal 73 to the memory reading circuit 8. The memory reading circuit 8reads the image signal from the frame memory 6 by offsetting theaddress, based on the received signal. Thereby, the entire screen isshifted.

The area discrimination circuit 72 sends the area signal 74 and themovement vector 75 within the area to the parameter setting circuit 9,and the parameter setting circuit 9 sets a different filtercharacteristic for each area to the filter 10. The area signal 74 isalso sent to the switch 76, which sends the image signal from the framememory 76 to the D/A converter 11 for the stationary area or to thefilter 10 for the moving area. The output of filter 10 is subsequentlysent to the D/A converter 11. That is, only for the moving area is thefiltering processing performed.

Note that the switch 76 can be integrated into the filter 10 by using afilter which allows the transmission of a whole frequency band, if thestationary area is considered as a special case in the filter 10.

The analog signal output of the D/A converter 11 has a synchronismsignal added by the synchronism signal adding circuit 12, and is sentout as the output video signal 77 being processed for each area.

FIG. 8 is a plan view showing images containing a plurality ofmovements. In this figure, 81 is an output screen which corresponds to amonitor display screen.

82 and 83 are first and second images of a previous screen,respectively. 84 and 85 are first and second images of a current screen,respectively. 86 is a background image composed of small squaresarranged in FIG. 8.

In transferring from the previous screen to the current screen, thefirst and second images 82, 83 of the previous screen are moved to thefirst and second images 84, 85 of the current screen on the outputscreen 81, respectively. However, the moving directions and sizes of thetwo images are different. The background image 86 is not moved herein,i.e., coincident between the previous screen and the current screen.

FIG. 9 is a plan view showing the area discrimination result.

91 and 92 are first and second moving areas, respectively. 93 and 94 arethe movement vectors within areas of the moving areas 91, 92. 95 is astationary area.

The area discrimination circuit 72 of FIG. 7 performs the area divisionand the calculation of the movement vector within the area as shown inFIG. 9. That is, the first and second moving areas 91, 92 and thestationary area 95 are divided, and the movement vectors 93, 94 withinthe areas are obtained for the first and second moving areas 91, 92.

When the device as shown in FIG. 7 is a tracking device, the tracking ofa specified image is performed. If it is assumed that the second image83 is tracked in the current screen, the area discrimination circuit 72sends the movement vector 94 within the area of the second moving area92 to the memory reading circuit as an address offset signal 73. At thistime, as a result of shifting of the image due to the offset applied inreading the image signal from the frame memory 6, the second image 84 ofthe current screen is displayed at the same place as the first image 83of the previous screen on the output screen 81. The images of otherareas are shifted.

The switch 76 sends the image signal of the first and second movingareas 91, 92 to the filter 10, and that of the stationary area 95directly to the D/A converter.

The filter 10 has its filter set with a different characteristic by theparameter setting circuit 9, depending on the movement vectors 93, 94within the areas, for the images of the first and second moving areas91, 92, in order to perform the filtering processing. The settings ofthe filter and parameters used in the filter 10 are the same as in thefirst embodiment.

As described above, the movement detection device of the presentinvention can compensate for the degradation of image quality resultingfrom the movement of the image with filtering processing, so that thereis an effect of increasing the resolution of the output image with themovement correction device, thereby providing a high quality image.

Further, the filtering characteristic useful for the filteringprocessing has such an effect that the image quality is made excellent,nearly optimal, by changing it adaptively based on the movement vectorto be obtained especially for the movement correction, and withoutincreasing the cost.

Next, to accomplish the sixth and seventh objects of the presentinvention, an embodiment in which the movement detection device isapplied to a focus detection apparatus to improve the focus detectionaccuracy will be described.

Recently, image equipment such as a video camera or an electronic camerahave been remarkably developed, and it is a requisite for its functionto have an automatic focus adjustment device.

By the way, for the focus detection device, there are provided a deviceof the passive type in which the focusing signal is obtained by takingthe correlation of the image picked up by a twin-lens optical system, oran automatic focus adjustment device of the active type in which thefocusing is judged from a position of a spot generated by reflectedlight flux by radiating an infrared light onto an object.

On the other hand, in the pick-up device such as a television camera, afocus detection device has been developed in which the focus detectionis performed by carrying out the image processing of the image signal.In such a device using image processing, the signal for detecting thefocusing state is obtained from the image signal, whereby there is afeature that the focus detection is allowed irrespective of the distancefrom an object, without special elements or a circuit for providing theinfrared projection, and with high precision, so that its developmenthas rapidly progressed.

FIGS. 16A-16B are views for explaining an embodiment of a focusdetection method with conventional image processing, illustrating theintensity distribution for the edge portion of an object image in theunfocused and focused states, in which FIG. 16A shows the unfocusedstate and FIG. 16B shows the focused state.

In FIG. 16A, EO shows an intensity distribution of the edge portion foran object image in the unfocused state, with a vague distribution due tothe unfocused condition and a large width of the edge portion.

Also, in FIG. 16B, EI shows an intensity distribution of the edgeportion for an object image in the focused state and at the same placeas for the intensity distribution EO of an unfocused edge portion. Inthe focused state, it shows a narrow and steeply rising-up edge.

Accordingly, the width of the edge portion for the object image isdetected, and focusing and unfocusing are judged from this edge width.That is, focusing can be judged by making use of the property of anarrow edge width.

The edge intensity distribution EO in the unfocused state is adistribution of the edge portion detected from the image signal, wherethe edge width is represented by the following expression.

    l.sub.1 =d.sub.1 /(dI.sub.1 (x)/dx)                        (6)

Where d₁ indicates an intensity difference of the edge portion. I₁ (x)is a function for representing the intensity distribution of the edge inthe unfocused state, whereby Di₁ (x)/dx indicates a slope of the edge.This slope of the edge can be used by taking the average of the slopesof focusing in a range from a portion where the edge rises up to aportion where it becomes flat again.

Also, when focused, the width of edge l₂ can be calculated from theintensity distribution EI for the focused state, using the followingexpression.

    l.sub.2 =d.sub.2 /(Di.sub.2 (x)/dx)                        (7)

Where d₂ is an intensity difference of the edge portion, I₂ (x) is afunction for representing the intensity distribution El of the edge inthe focused state, and Di₂ (x)/dx is a slope of the edge. d₂ has almostthe same value as d₁, and Di₂ (x)/dx is larger than Di₁ (x)/dx.

Accordingly, since l₂ is a smaller value than l₁, it can be seen thatthe edge width becomes smaller and the focusing has been adjusted.

In this way, generally, a method (in which the edge width is calculatedfrom a density difference and a slope of the edge portion, and the stateis judged nearer to the focused state if the value is smaller) ispracticed as one method of performing focus detection with imageprocessing.

However, in the above-mentioned focus detection method, when an imagehas movement, i.e., when an object is moving, or when the whole screenhas shifted as a result of trembling of the hand and/or panning, theimage becomes unfocused due to the movement, whereby there is adisadvantage that the width of the edge is widened due to the movement,and correct detection of the focus state can not be made.

The following embodiment has been made to resolve the above-mentionedproblems, and is characterized by focus detection apparatus comprisingmovement detection means for detecting a movement of an image from theimage pick-up signal, movement correction means for correcting themovement of the image based on the output of the detection means, andfocus detection means for performing the focus detection by extracting asignal component which varies with the focusing state from the pick-upsignal having its movement component corrected by the movementcorrection means.

Thereby, a high precision focus detection apparatus can be realizedwithout decreasing its precision owing to the movement of the image, inwhich the focus detection can be performed by detecting the movement ofan image from an image signal, and using the signal for which thedegradation of resolution due to the movement of the image iscompensated.

A focus detection apparatus of this example will be described in detailbelow with reference to the drawings.

FIG. 10 is a block diagram showing a third example in which a movementdetection device of the present invention is applied to the focusdetection apparatus. In the same figure, 101 is an object, 102 is apick-up lens, and 103 is a pick-up element (or pick-up tube) such as aCCD for outputting a pick-up signal by photoelectrically converting anobject image formed on a pick-up plane by the pick-up lens 2. 104 is anamplifier for amplifying the pick-up signal output from the pick-upelement 3 to a predetermined level, 105 is an A/D converter forconverting an input analog pick-up signal to the digital signal, 106 isa movement vector operation circuit for obtaining the movement vector ofthe image from the image signal which has been converted to the digitalsignal by the A/D converter 105, 107 is an x-axis projection circuit forprojecting the movement vector on the x-axis that is a horizontaldirection of the screen, and 108 is an x component of the movementvector.

109 is an edge detection circuit for detecting the width of the edgeportion for an object image, 110 is an edge width detection circuit fordetecting the width of the edge portion detected by the edge detectioncircuit 109, and 111 is an edge width signal.

112 is a comparator, and 113 is a memory. 114 is a lens control circuit,115 is a lens control signal, and 116 is a lens driving circuit formoving the pick-up lens 102 in a direction of the optical axis to adjustthe focus.

On the other hand, 117 is a video signal processing circuit foroutputting a standard television signal by performing signal processingsuch as gamma correction or various filtering of the pick-up signaloutput from the A/D converter 105, 118 is a D/A converter for convertingthe digital signal output by the video signal processing circuit 117 toan analog signal, and 119 is a synchronism signal adding circuit foradding a synchronism signal to the image signal output from the D/Aconverter 118, and 120 is an output video signal.

With the above constitution, the pick-up lens 102 forms an image ofobject 101 on the pick-up plane of pick-up element 103, and the pick-upelement 103 converts photoelectrically the image of the object to outputan image signal. The image formed on the pick-up element 103 includesmovement owing to the movement of the pick-up lens 102, pick-up element103, or object 101.

The image signal output from the pick-up element 103 is amplified by theamplifier 104, and converted to a digital signal by the A/D converter105.

A part of the digitized image signal is input into the movement vectoroperation circuit 106. The movement vector operation circuit 106contains a frame memory where the image of a previous frame is stored,and in which the movement vector of the image is calculated by comparingthe image of the current frame with that of the previous frame stored inthe frame memory. Exemplary of the method of calculating the movementvector is that used in a so-called representative point matching method,or a gradient method can be used.

The x-axis projection circuit 107 gives an x-axis component signal 108from the movement vector obtained by the movement vector operationcircuit 106.

The digitized image signal is sent to an edge portion detection circuit109. The edge portion detection circuit 109 detects an edge portion fromthe information such as a slope for the image signal, and selects anedge having the largest slope with respect to the x-axis, for example,as the edge for judgement of the focusing. The edge width operationcircuit 110 calculates the width of the edge from a density differenceof the edge and a slope of the edge, as above-described, and outputs itas an edge width signal 111. The edge width signal 111 is subtractedfrom the x-axis component signal 108 of the movement vector and thensent to the comparator 112. The signal sent to the comparator 112 is anedge width signal with the movement of the image corrected. And thecomparator 112 compares the input corrected edge width signal with datawithin the memory 113.

Within the memory 113, the corrected edge width signal in the previousfield or frame is stored.

The comparator 112 sends a control signal to the lens control circuit,as well as writing a smaller edge width signal of two corrected edgewidth signals into the memory 113.

The control signal is one in which driving of the current pick-up lens102 is continued in the same direction when a newly input edge widthsignal is smaller than the previous edge width signal, or driving isperformed in a reverse direction when it is larger than the previousedge width signal. Also, it may be permitted to send a signal in whichthe amount of driving the lens is gradually changed depending on theamount of variation in the corrected edge signal. Or when the amount ofvariation is quite small, a control signal for stopping the driving ofthe lens can be sent by deciding on the focusing state if the amount ofvariation is changed from negative to positive. In order to make thefine control, it is desirable that corrected edge signals (as much asseveral frames) should be stored in the memory 113.

The lens control circuit 14 issues a lens driving signal 115, based on acontrol signal sent from the comparator 112, in accordance with whichthe lens driving circuit 116 drives the pick-up lens 102.

On the other hand, the digitized image signal is input into the videosignal processing circuit 117, is then input into the D/A converter 118for conversion into the analog signal, has a synchronism signal added inthe synchronism signal adding circuit 119, and is output as a videosignal.

FIG. 11 is a view showing schematically the movement of the image, likein FIG. 2. In this figure, 121 is an output screen. For example, amonitor display screen can be assumed. Like in FIG. 2, 112 is a previousimage, and 123 is a current image. 124 is a movement vector when theprevious image 122 moves to the current image 123. 125 is a projectionvector of the movement vector 124 onto the x-axis.

In the output screen 121, a video signal 120 of FIG. 10 is displayed.

For example, an image displayed with trembling of the hand or panningshows that movement from previous image 122 to current image 123 whentime has passed from one clock to another even if the object 101 is astationary object.

Accordingly, the movement vector 124 is calculated in the movementvector operation circuit 106, and the projection vector 125 iscalculated in the x-axis projection circuit and output as an x-axissignal component 108.

Here, as the current image 123 includes movement, the value of eachpixel is integrated in the direction of movement vector 124.Accordingly, in practice, current image 123 as shown in FIG. 11 onlyshows approximately a gravitational center position of the sides of eachpattern.

FIG. 12 is a front view showing the movement of the optical system. Inthe same figure, 131 is one cell when the pick-up element is asolid-state pick-up element such as a CCD.

For the convenience of explanation, it is assumed that the movementdirection of the image coincides with the array direction of one cell inthe pick-up element 103. The pick-up element 103 converts a pattern ofthe optical image thereon during a predetermined period of exposure. 132is an optical image at the start of exposure, and 133 is an opticalimage at the termination of exposure. The optical image 132 at the startof exposure has been moved to the optical image 133 at the terminationof exposure because the image moves during exposure like in FIG. 11. Themovement vector 124 as shown in FIG. 12 indicates the movement betweenthe optical image 132 at the start of exposure and the optical image 133at the termination of exposure. This is almost the same as in FIG. 11,but more strictly, slightly different. That is, the movement vector asshown in FIG. 11 is one taken at nearly intermediate times of respectiveexposure periods for two images. On the contrary, the movement vector asshown in FIG. 12 is one taken for an image of interest from the start ofexposure to the termination of exposure.

Accordingly, when the amount of image movement changes abruptly, boththe movement vectors may have different values. Generally, both valuescan be almost the same, but the movement vector 24 can be used afterslight correction, because the movement vector 124 obtained from themovement vector operation circuit 106 is as shown in FIG. 11.

FIG. 13 is a characteristic graph showing a point image distributionfunction with the movement of the image. The form of the function is thesame as the example of FIG. 4. 141 is a point image distributionfunction. Here, it is assumed that the length of projection vector 125is c. Then, the line generated by one point of object 102 moves duringthe exposure period, approximately following the point imagedistribution function 141. Representing that function as h(x) which canbe represented as previously shown in the expression (1),

    h(x)=Rect(x/c)                                             (8)

The pick-up image becomes unfocused in accordance with the expression(1), due to its movement as well as the out-of-focus state.

FIG. 14 is a characteristic graph showing the correction of the edgewidth.

151 is an image signal of the edge portion, with its intensitydistribution being I(x). This corresponds to a signal directly obtainedfrom the pick-up element 103. 152 is an intensity distribution of theedge portion when the image does not contain any movement, its intensitydistribution being I'(x). The dullness of the edge for I'(x) can bedetermined by the shape of an original object and the amount ofout-of-focus.

The image signal 151 for the edge portion is an image signal of theportion selected in the edge portion detection circuit 109. The edgewidth operation circuit 110 calculates the width of edge 1 from adensity difference of edge d and a slope dI(x)/dx, such that

    l=d/(dI(x)/dx)                                             (9)

However, l contains the amount of unfocus due to the movement of theimage.

Between I(x) and I'(x), the following relation stands.

    I(x)=T'(x)*h(x)                                            (10)

Here, * represents the convolution operation.

h(x) is a function of width c as shown in FIG. 13, in which I(x) has anedge slope portion widened by about c/2 to the right and left, comparedwith I'(x), as shown in FIG. 14. Accordingly, assuming the width of theedge for I'(x) as I', the following expression approximately stands.

    l=l'+c                                                     (11)

The width of edge l directly obtained from the image signal 151 of theedge portion contains the size c of projection vector 125, thus having alarger value than originally obtained, and may vary in accordance withthe movement of the image. In the expression (11), by subtracting c froml, it is clear that l' can be obtained, and is not subject to theinfluence of the movement of the image by judging the focusing based onl'.

That is, as shown in FIG. 10, by subtracting an x-axis signal component108 having the value of c from an edge width signal 111 having the valueof l, l' is obtained, and then input into the comparator for use as asignal for judging the focusing, so that the detection of focusing canbe made without influence of the movement of the image.

In the explanation given herein, by considering that the intensitydistribution of the edge portion is obtained along the x-axis, themovement vector is also projected onto the x-axis. As the movementvector 24 is obtained in the form of a representation with x-axis andy-axis components, the processing for the x-axis projection ispractically unnecessary. But if the intensity distribution of the edgeportion is obtained in a horizontal axis, i.e., other than x-axis, forexample, in the steepest direction of the edge portion of interest, soas to calculate the width of the edge, the projection of the movementvector is necessary. The size of the projection vector can be obtainedin a simple manner as a value of the inner product between a unit vectorin its direction and the movement vector.

FIG. 15 is a block diagram showing a fourth example of focus detectionapparatus according to the present invention. In the same figure, 161 isa frame memory, and 162 is a filter. 163 is parameter setting circuitfor supplying parameters to the filter 162. 164 is a filtered imagesignal.

In this example, the signal read from the pick-up element 103 is passedto the filtering processing based on the movement vector, to perform thefocus detection from that signal.

Like in the third embodiment, the digitized image signal is output fromthe A/D converter 105. The frame memory 161 stores this signaltemporarily.

The digitized image signal is also sent to the movement vector operationcircuit 106, where the movement vector 124 is calculated. The framememory 161 acts as a delay element for synchronizing the image signal toterminate the calculation of movement vector 124.

The signal read from the frame memory 161 is sent to the filter 162,where the filtering processing is performed. The frequencycharacteristic of filter 162 is set by the parameter setting circuit163, where the parameters are given to the filter 162 so as to correctthe unfocused image from the calculation result of movement vector 124.

The filtered image signal 164 is sent to the edge portion detectioncircuit 109, where like in the first embodiment, the edge portion isselected, and after that operation, the focusing judgment operation isperformed to drive the pick-up lens 102.

On the other hand, the filtered image signal 164 is sent to the videosignal processing circuit 117, where like in the third embodiment, it isfinally output as an output video signal 120.

The principle of this example will be described below.

Let the size of movement vector 124 be a', and let the spatialcoordinate along the movement direction be x'. The point imagedistribution function h'(x) generated by the movement vector 24 is givenby, in accordance with the previous expression (1),

    h'(x)=Rect(x'/a)                                           (12)

The degradation of the image can be corrected by filtering.

The filter for the filtering can use the characteristic as shown inFIGS. 4B and 4C in the first example. Explaining this using FIGS. 4B and4C, 42 is a frequency characteristic showing the degradation of theimage with movement, and can be represented by the previous expression(2).

Accordingly, the frequency characteristic of the inverse filter forcompensating for the degradation of the image as described above is asshown by 43 in FIG. 4C. That is, H(f)·P(F)=1, in which the filteringwith P(f) allows the compensation for the degradation of the image withH(f).

By filtering with that filter in the filter 162 in FIG. 15, theresolution of output video signal 120 is increased, so that an excellentimage can be obtained. However, in this embodiment, the inverse filtercan be achieved only approximately, because it becomes infinite at thefrequency where H(f) is zero. The range of the frequency can be a rangewhere the frequency spectrum for the image signal exists.

As clearly seen from FIGS. 4A and 4C, expressions (2) and (3), thecharacteristic of degradation H(f) and the characteristic ofcompensation filter P(f) contain the size a of movement vector 24 as aparameter. The x'-axis and f-axis are taken along the direction ofmovement vector 124, respectively, and P(f) depends on the size anddirection of movement vector 124.

Accordingly, it is desirable that the filter 162 changes adaptively inaccordance with the movement vector 124.

In order to perform the filtering processing in the filter 162, thereare two methods like in the previous first embodiment. One of them isthe filtering on the axis of frequency, in which the Fourier transformis taken of the image signal read out from the frame memory 161 with anFFT (Fast Fourier Transform), which is then multiplied by the inversefilter P(f), and the inverse Fourier transform is taken, so that thefiltered image signal is obtained.

Another method is the filtering on the axis of time, performed in such away that the impulse response is obtained by the inverse Fouriertransform of the inverse filter P(f), and convoluted to an image signalfrom the frame memory 6 so that a filtered output can be obtained.

As regards the constitution of the filter and the filtering processingused herein, the inverse filter and the Wiener filter can be used, asdescribed using expressions (10), (11) and FIGS. 5, 6 in the previousfirst embodiment, and the explanation of the analysis is omitted. Withthe filtering, the degradation of the image, i.e., the degradation ofresolution, can be compensated, so that an excellent focus detection canbe made.

Also, in the filter 162, other various filters may be used. As thefrequency characteristic 43 has a characteristic of a low-pass filter,the filter characteristic of filter 162 has generally a characteristicof a high-pass filter or a band-pass filter.

The edge portion of the edge signal in the filtered image signal 164 isrepresented as a signal analogous to that as shown in FIG. 14. That is,before filtering, it is like the edge portion of image signal 151, butafter filtering, it becomes the intensity distribution 152 when there isno movement of the image.

Accordingly, by calculating an edge signal width from the filtered imagesignal 164 and using it for judgment of the focusing, a stable detectionof the focus can be performed without almost any influence of imagemovement. Also in the case, the value of the edge width obtained issmaller than if the edge width is calculated without taking intoconsideration the movement of the image.

Also, in this embodiment, there is a feature that the output videosignal 120 itself can provide an excellent and high-quality image withhigh resolution.

When the whole screen does not move and an object is moving, by dividingthe area, portions having common movements can be subjected to theprocessing based on the same movement vector.

It is noted that for the detection of image movement, the method ofprocessing the image signal was described, but when the servicecondition of the pick-up device is subject to much vibration, andprovided with a vibration-proofing apparatus, the movement vector signalcan be obtained from an angular sensor or acceleration sensor of avibration-proof apparatus. In this case, it is not simple to deal withthe movement of the object, but when the movement of the whole screen ispredominant due to the vibration, a desired movement vector can beobtained and excellent focus detection can be made.

As above described, with the focus detection apparatus according to thepresent invention, focus detection can be performed by detecting asignal component varying with the focusing state from the pick-upsignal, wherein movement correction means is provided for detecting andcorrecting the movement of the image. The focus detection is made basedon a signal component having the movement corrected by said movementdetection means, so that it is possible to prevent the degradation ofaccuracy in the focus detection means caused by the decrease of signaldepending on the focusing state, such as the dullness at the edgeportion or the decrease of the high frequency component due to themovement of image. Thus, a stable focus detection apparatus with highprecision can be provided at all times and it is not subject to theinfluence of camera vibration or the movement of the object.

We claim:
 1. A movement detection device for detecting movement of animage incident on an image sensing device, and for compensating for themovement of the image, comprising:(A) movement detection means fordetecting the movement of the image from image signals output from theimage sensing device; (B) correction means for performing an electricalmovement correction on the image signals output from said image sensingdevice according to the movement detected by said movement detectionmeans; and (C) filter means for receiving the movement-corrected imagesignals output from said correction means, and for filtering themovement-corrected image signals to compensate for degradation of imageresolution due to the movement of said image.
 2. A movement detectingdevice according to claim 1, wherein a filter characteristic of saidfilter means is adaptively determined in accordance with the movement ofsaid image.
 3. A movement detection device according to claim 2, whereinsaid filter means comprises an inverse filter having a frequencycharacteristic for correcting a frequency characteristic of the movementof image.
 4. A movement detection device according to claim 2, whereinsaid filter means comprises a Wiener filter.
 5. A movement detectiondevice according to claim 2, wherein said movement detection meansdetects a movement vector of the image from the image signals outputfrom the image sensing device.
 6. A movement detection device accordingto claim 5, wherein said movement detection means changes the filtercharacteristic adaptively in accordance with the detected movementvector.
 7. A movement detection device according to claim 1, furthercomprising movement correction means for correcting the movement of theimage based on the output of said movement detection means.
 8. Amovement detection device according to claim 7, further comprising animage memory for storing the image signals, and wherein said movementcorrection means corrects the movement of the image by changing anaddress for reading the image signals from said image memory based onthe amount and direction of detected image movement.
 9. A movementdetection device according to claim 8, wherein said filter means iscoupled to an output side of said image memory.
 10. A movement detectiondevice comprising:(A) image pick-up means for picking up an image andoutputting image pick-up signals corresponding thereto; (B) an imagememory for storing the picked-up image signals; (C) movement vectordetection means for detecting a movement vector of the image using saidimage pick-up signals; (D) image movement correction means foroffsetting the movement of the image by changing a read address forreading the picked-up image signals from said image memory in accordancewith said movement vector; (E) filter means for filtering said imagepick-up signals read out from said image memory by said image movementcorrection means; and (F) correction means for correcting a degradationof image resolution caused by movement of said image by changing afrequency characteristic of said filter means, based on the movementvector output by said movement vector detection means.
 11. A movementdetection device according to claim 10, wherein said filter means has afrequency characteristic for compensating for the frequencycharacteristic of the image movement.
 12. A movement detection deviceaccording to claim 11, wherein said filter means comprises an inversefilter of inverse polarity to the frequency characteristic of said imagemovement.
 13. A movement detection device according to claim 11, whereinsaid filter means comprises a Wiener filter.
 14. A movement detectiondevice for detecting movement in an image from image signals output froman image sensing device, said movement detecting device comprising:(A)movement detection means for detecting movement of an image within animage screen from the image signals output by said image sensing device;(B) movement correction means for correcting the movement of the imageby correcting the image signals output from said image sensing devicebased on an output of said movement detection means; (C) filter meansfor performing a filter processing operation on the corrected imagesignals output from said movement correction means to compensate for thedegradation of image signal resolution due to the movement of saidimage, based on the output of said movement detection means; and (D)filter control means for adaptively controlling an operation of saidfilter means on the basis of the output of said movement detection meanswhen said movement detection means detects at least one of a moving areaand a stationary area in said image screen.
 15. A movement detectiondevice according to claim 14, wherein said filter control means controlssaid filter means to perform filter processing on the image signals inthe moving area within the image screen, when said movement detectionmeans detects said moving area.
 16. A movement detection deviceaccording to claim 15, wherein a filter characteristic of said filtermeans is adaptively changed by said filter control means in accordancewith the movement of said image.
 17. A movement detection deviceaccording to claim 14, wherein said filter means comprises an inversefilter having a frequency characteristic for correcting a frequencycharacteristic of the movement of the image.
 18. A movement detectiondevice according to claim 14, wherein said filter means comprises aWiener filter.
 19. An image processing device for detecting movement ofan image from a received image signal, and for compensating for themovement of the image, comprising:(A) image signal receiving means forreceiving the image signal; (B) movement detection means for detectingthe movement of the image from the image signals received by said imagereceiving means; (C) compensating means for performing an electricalmovement compensation on the image signals output by said image signalreceiving means according to the movement detected by said movementdetection means; and (D) filter means for receiving themovement-compensated signals output from said compensating means, andfor filtering the movement-compensated image signals to improve an imagequality of the movement compensated signals.
 20. A device according toclaim 19, wherein said filter means comprises an inverse filter having afrequency characteristic for compensating a frequency characteristic ofthe movement of the image.
 21. A device according to claim 19, whereinsaid filter means comprises a Wiener filter having a filtercharacteristic for compensating a frequency characteristic of themovement of the image.
 22. A device according to claim 19, wherein saidmovement detection means detects movement vectors of the images from theimage signals received by said receiving means.
 23. A device accordingto claim 22, wherein said compensating means comprises:an image memoryfor storing the image signal received by said receiving means; a memorycontrol circuit for controlling an address for reading out the imagesignal from said image memory to compensate the movement of the imageaccording to the movement vectors detected by said movement detectionmeans.
 24. A device according to claim 19, wherein a frequencycharacteristic of said filter means is adaptively determined inaccordance with the movement of the image detected by said movementdetection means.